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Executive Summary 


A bipartisan experiment in local flexibility spanning several federal agencies, called Performance Partnership 
Pilots (P3) for Disconnected Youth, may need more time for local, state, and tribal projects to fully test waivers of 
federal legislative, regulatory, and administrative barriers that may be hindering their performance. 


First authorized by Congress in January 2014, P3 allows a limited number of local projects that are focused on 
youth to align and better coordinate federally-funded programs and services. The program permits projects to 
request waivers of certain federal restrictions and to potentially consolidate multiple funding streams. 


Although focused on a particular subset of youth — those who are disconnected or at risk of becoming 
disconnected from education or the workforce — the program is seen as a possible model for increased federal 
flexibility, with broader potential implications for other social programs nationally. Initial participating federal 
agencies included three departments and two agencies: the Departments of Education, Labor, and Health and 
Human Services, the Corporation for National and Community Service, and the Institute of Museum and Library 
Services. Congress has since expanded the initiative to include programs at two other federal departments, the 
Department of Justice and HUD. 


So far, Congress has authorized three rounds of up to 10 pilots each. The administration announced a first round 
of nine awards in October 2015. 


This paper examines the early progress of this initiative. It is based on a review of project grant applications and 
performance agreements, interviews with six of the first round pilots, and interviews with federal officials 
overseeing the program. Highlights of the paper’s findings include the following: 


e Collaboration: The P3 program is, in part, an effort to promote coordinated services for disconnected 
youth. All of the first round pilots include multiple local partners, often including schools, local workforce 
programs, other public agencies, nonprofit social service providers, and outside researchers. 


While it is still early, all of the P3 pilots feature most of the attributes commonly associated with 
successful collaborative efforts, including common agendas, shared measurement, and backbone 
support. P3 start-up funding has also played an important catalytic role, gap-filling on management and 
evaluation costs. All of the interviewed pilots said that if they produce results they will continue their 
efforts after their P3-funded start-up phase ends. 


e Flexibility: Despite the emphasis on collaboration, P3’s flexibility provisions are what make it unique. 
However, while this may change, most of the waivers approved thus far for the first round of pilots 


1 For more information, contact Patrick Lester, Director, Social Innovation Research Center, at (443) 822-4791 or 
patrick@socialinnovationcenter.org. 


appear to be modest. 


Most pilots have no more than two approved waivers, with most involving what appear to be minor 
changes in program eligibility or reporting requirements. Only four of the nine pilots have been granted 
authority to blend federal funds. 


The slow start on local flexibility appears to be due to a combination of factors, including limited time, 
limited local knowledge of federal barriers, complications attributable to recent changes in federal labor 
and education laws, the existence of state and local barriers that P3 does not address, and legislated 
safeguards that were intended to protect vulnerable populations. 


Most of these barriers seem likely to be overcome over time. The first round pilots may apply for 
additional waivers in the coming months and years. Federal officials are providing technical assistance. 
Second and third round pilots are also likely to draw lessons from the earlier cohorts. 


e Data Systems: Interoperability among multiple education, workforce, and other data systems is a 
central issue for the P3 program. However, P3 start-up grants are insufficient to fully cover these costs. 
Although the pilots have brought some pre-existing data capacity to their projects, additional capacity will 
probably be needed in many cases if the projects are to be sustained or scaled after their pilot phase 
ends. 


e Evaluations: Evaluations, both local and national, are a significant component of P3. However, 
expectations for quick results should be tempered by the relatively nascent state of evidence for 
programs for disconnected youth generally and for multi-agency strategies like P3 in particular. 


Some of the projects, particularly those that are well-grounded in existing evidence and/or existing 
programs, may produce significant results. Other evaluations may be more interim in nature, reflecting 
projects that need more time to reach their potential. 


The national evaluation is also not designed to draw definitive and generalizable conclusions about the 
direct impact of the P3 program on disconnected youth, at least in the near term. It will instead focus on 
the removal of barriers and improvements in system coordination. 


e Sustainability: Waivers for the first round pilots are scheduled to expire on September 30, 2018. 
Authority for second and third round pilots will expire in 2019 and 2020, respectively. The literature on 
collaborative projects like P3 strongly suggests that a longer-term commitment is needed to be 
successful. 


While the local pilots have said that they will continue with existing and/or alternative funding, the 
flexibility provisions will expire when the pilot phase ends. With local pilots still testing the limits of their 
new authority, Congress should consider extending these deadlines. 


This paper reviews each of these issues in greater detail and concludes with recommendations. 


Introduction 


At its heart, the P3 program is a test of several interrelated strategies. It is partly a test of providing increased 
flexibility to states, localities, and tribes in the provision of federally funded education, workforce, and social 
services. It is also partly a test of collaboration as a strategy to bring together multiple programs and providers 
with a common focus on populations with complex needs. 


As described in federal grant documents, it is intended to “test the hypothesis that additional flexibility for states, 
localities, and tribes, in the form of blending funds and waivers of certain programmatic requirements” will result 


in improved outcomes for social service recipients @ — in this case, disconnected youth aged 14-24 who are low 
income and either homeless, in foster care, involved in the juvenile justice system, unemployed, or either not 
enrolled in or at risk of dropping out of an educational institution.? 


By itself, such flexibility offers significant potential, opening possibilities for increased innovation and the 
provision of services that are better suited to local environments or the needs of specific populations. However, 
P3’s increased flexibility is also designed to facilitate a second complementary strategy: greater cooperation and 
collaboration among multiple services and programs with shared goals. According to the 2014 grant application 
materials, it is intended to address: 


[P]oor coordination and alignment across the multiple systems that serve youth; policies that make it 
hard to target the neediest youth and help them overcome gaps in services; fragmented data systems 
that inhibit the flow of information to improve results; and administrative requirements that impede 
holistic approaches to serving this population. 


Finally, P3 is a test of the idea that compliance-based regulatory accountability can be reduced in favor of 
increased performance-based accountability. It is a bet that providing increased flexibility on certain legislative, 
regulatory, and administrative restrictions and better aligning services will produce better outcomes for affected 
youth. 


How well are these ideas working? With the first round of P3 projects just underway, it is too soon to know with 
certainty. Nevertheless, enough work has been done at both the national and local levels to provide an early 
indication of progress and emerging challenges. 


Methodology 


This paper is based on several sources of information. They include a review of grant applications and 
performance agreements for all nine of the first round P3 pilots, which were obtained through public records 
requests or directly from the pilots.4 Interviews were conducted with six of the nine pilots, officials at three of the 
participating federal agencies (the Departments of Education, Labor, and Health and Human Services) and the 
White House Office of Management and Budget (OMB), and other outside experts.® 


Except where views are attributed by name, the opinions expressed in this paper are not necessarily endorsed 
or shared by these individuals or organizations. 

Background 

Fully understanding the P3 program requires knowing both its history and the broader political context. This 


section reviews the thinking and efforts that shaped the initiative, including administration efforts and 
congressional authorization. It concludes with a brief summary of the first round pilots. 


2 U.S Department of Education, “Applications for New Awards; Performance Partnership Pilots,” Federal Register, November 24, 2014. 
Available at: https://www.federalregister.gov/articles/2014/11/24/2014-27775/applications-for-new-awards-performance-partnership-pilots 

3 PL 113-76, Division H: Sec. 526(a)(2) 

4 All of the grant applications and most of the performance agreements were obtained through two Freedom of Information Act (FOIA) 
requests submitted to the U.S. Department of Education. The first request for grant applications (Number 16-00300-F) was submitted on 
November 8, 2015 and completed in full in January, 2016. The second request for performance agreements (Number 16-00894-F) was 
submitted on February 17, 2016 and initially rejected on February 25, but later fulfilled after appeal on April 6, 2016. Seven of the nine 
performance agreements were obtained through this FOIA request. The other two were obtained from the pilots. 

5 Interviews with six of the nine pilots were conducted from February 8 to February 21, 2016. Three pilots declined to participate (Baton 
Rouge, Seattle-King County, and Ysleta del Sur Pueblo). Interviews with federal officials were conducted from March 1, 2016 through 
April 5, 2016, with subsequent communications focused on comments and proposed corrections to early drafts of this paper. 
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The Early Roots of P3: Collaboration and Flexibility 


P3’s roots can be traced back to several earlier initiatives that emphasized flexibility or increased coordination of 
youth programs. One early effort that also supported collaboration and flexibility was an initiative developed 
during the administration of President George W. Bush called Helping America’s Youth.® Led by First Lady Laura 
Bush, the initiative laid the groundwork for the creation of the Interagency Working Group on Youth Programs, a 
broad federal coordinating body.’ Similar coordinating bodies at the state level, called children’s cabinets, also 
began appearing or gaining momentum during this period.® 


Collaborative neighborhood-focused initiatives launched in the early years of the Obama administration were 
also influential. These included Promise Neighborhoods, a cradle-to-college neighborhood-focused program at 
the Department of Education that was inspired by local initiatives like the Harlem Children’s Zone in New York 
City and the Strive Network in Cincinnati.? Other neighborhood-focused programs, such as Choice 
Neighborhoods at HUD and the Byrne Criminal Justice Innovation program at the Department of Justice, also 
emphasized collaborative, data-driven services in low-income communities. ' 


The White House Council for Community Solutions, a panel of experts appointed by the president in late 2010, 
also played a contributing role. It also focused on local collaborative efforts, eventually turning its attention to the 
needs of disconnected youth.'' With the assistance of the Bridgespan Group, the council reviewed multiple local 
efforts and concluded that while most failed to achieve significant results, those that were successful tended to 
share certain characteristics, including: effective leadership, shared vision and collaboration, dedicated staff 
capacity, sufficient funding, long-term commitment, and the use of data to monitor improvement over time.'2 


The most important driver, however, was an Obama administration focus on increasing state, local and tribal 
flexibility. In 2011, the administration issued a presidential memo that directed federal agencies to work with 
these governments to identify administrative, regulatory, and legislative barriers that hindered performance.'? 
OMB provided additional details in a subsequent memo that directed federal agencies to assemble flexibility 
plans that included many of the elements that would eventually define P3.1* When the Obama administration 
solicited ideas for likely programs to test these concepts, stakeholders pointed to disconnected youth. 


Establishing P3 


Flowing from these earlier efforts, the administration began work on what would eventually become P3. In its 
proposed FY 2013 budget, submitted to Congress in February 2012, the administration requested authority for 
the Departments of Labor, Education, and Health and Human Services for pilot programs that would grant 
flexibility to a select group of states and localities in exchange for strong accountability measures. 


In March 2012, following the submission of its budget proposal, the administration created an Interagency Forum 
on Disconnected Youth, which began planning how to structure the new program. In June, the administration 


The White House, "Helping America’s Youth." Available at: http://georgewbush-whitehouse.archives.gov/firstlady/helping-youth.html 
The White House, Executive Order 13459 - Improving the Coordination and Effectiveness of Youth Programs, February 7, 2008. 
Available at: https://www.gpo.gov/fdsys/pkg/WCPD-2008-02-11/pdf/WCPD-2008-02-11-Pg163.pdf 

Forum for Youth Investment, "Children's Cabinet Network." For more information, see: http://forumfyi.org/childrens-cabinet-network-O 
Alliance for Children and Families, "Tipping Neighborhoods to Success," May 31, 2009. Available at: 
http://www.alliance1.org/sites/default/files/pdf_upload/report_pp/Tipping neighborhoods.pdf 

10 The White House, Office of Urban Affairs, "Neighborhood Revitalization Initiative." See: 
https://www.whitehouse.gov/administration/eop/oua/initiatives/neighborhood-revitalization 


The White House, Executive Order 13560—White House Council for Community Solutions, December 14, 2010. Available at: 
https://www.whitehouse.gov/the-press-office/2010/12/14/executive-order-13560-white-house-council-community-solutions 


White House Council for Community Solutions, "Community Solutions for Opportunity Youth: Final Report," June 2012. Available at: 
http://www.serve.gov/sites/default/files/ctools/12 O604whccs finalreport.pdf; See also: Civic Enterprises & America’s Promise Alliance, 
"Opportunity Road: The Promise and Challenge of America’s Forgotten Youth," January 2012. Available at: 
http://www.civicenterprises.net/MediaLibrary/Docs/opportunity_road.pdf 

The White House, “Presidential Memorandum — Administrative Flexibility,” February 28, 2011. Available at: 
https://www.whitehouse.gov/the-press-office/20 11/02/28/presidential-memorandum-administrative-flexibility 

Office of Management and Budget, Memorandum M-11-21, "Implementing the Presidential Memorandum ‘Administrative Flexibility, 
Lower Costs, and Better Results for State, Local, and Tribal Governments’," April 29, 2011. Available at: 
https://www.whitehouse.gov/sites/default/files/omb/memoranda/2011/m11-21.pdf 
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released a formal Request for Information that solicited stakeholder input.'> In March 2013, the administration 
released a paper summarizing the responses it received across several themes, including data, evaluation, 
outcomes measures, local partnerships, design issues, and possible subpopulations of particular concern.'® 


The next step was to formally create the P3 program. This required congressional approval, however, which was 
not assured. Increasing local flexibility is an idea that has drawn bipartisan support, but it has also been a 
source of partisan tension. Republicans often distrust authority that has been centralized at the national level 
and prefer that states and local governments be given more discretion to tailor programs to local needs. 
Democrats are often willing to allow limited local flexibility to achieve better outcomes, but they fear that local 
officials may use this authority to reduce or divert funding to other uses. 


Enacting P3 required finding a workable compromise between these two points of view. In its final form, P3 
piloted the bottom-up flexibility for states and communities that Republicans wanted, but it also subjected these 
flexibility requests to federal approval and compliance with civil rights laws and other protections to allay 
Democratic concerns.'” 


The proposal sidestepped a broader debate between the two parties over block granting federal programs, an 
idea that Republicans have supported as a way to shift power to states and local governments, '® but that 
Democrats have opposed for fear that such efforts would lead to spending cuts.'9 It avoided this debate in two 
ways: first by leaving the underlying federal programs and their associated funding levels unchanged, and 
second by excluding from its provisions any mandatory and other entitlement spending (e.g., Medicaid, SNAP, 
Temporary Assistance for Needy Families), which were the programs of most concern to Democrats. 


Over the course of the next year, the Forum for Youth Investment, a nonprofit organization that works with state 
children’s cabinets, lobbied Congress to include authorizing language in federal appropriations legislation. 
Congress did so and President Obama signed appropriations legislation into law with language creating P3 on 
January 17, 2014.20 


First Round Awards 


Once the initiative was formally enacted, the administration moved quickly to roll it out. In April, the 
administration released a consultation paper providing an outline of the new initiative.2’ The five initial federal 
agencies and departments affected by the legislation — the Departments of Education, Labor, and Health and 
Human Services, the Corporation for National and Community Services, and Institute of Museum and Library 
Services — signed an interagency agreement that governed their respective responsibilities.22 


1S One especially noteworthy response came from the Center for Law and Social Policy. See CLASP, "Comments to U.S. Department of 
Education Request for Information on Strategies for Improving Outcomes for Disconnected Youth," July 2012. Available at: 
http://www.clasp.org/resources-and-publications/files/CLASPCommentstoUSDOE-20120VAE0014.Final-2. pdf 

16 Interagency Forum on Disconnected Youth, “Summary of RFI Themes,” March 2013. Available at: 
http://youth.gov/docs/Summary%200f%20RF|I%20themes.pdf 

17 The program’s guardrails and other protections are discussed in more detail later in this paper. 

18 Social Innovation Research Center, "Performance and Outcomes in the Ryan Anti-poverty Plan," July 24, 2016. Available at: 
http://www.socialinnovationcenter.org/?p=96 

19 Jared Bernstein, "Block Grants Can be the ‘Poisoned Chalice’ of Social Policy," The Washington Post, March 28, 2016. Available at: 
https://www.washingtonpost.com/posteverything/wp/2016/03/28/block-grants-the-poisoned-chalice-of-social-policy/ 

20 PL 113-76, which covered funding for federal fiscal year 2014. The P3 authorizing language can be found in Division H: Sec. 526. 

21 Interagency Forum on Disconnected Youth, "Changing the Odds for Disconnected Youth: Initial Design Considerations for Performance 
Partnership Pilots," April 28, 2014. Available at: http://youth.gov/youth-topics/reconnecting-youth/performance-partnership- 
pilots/consultation 

22 According to interviews with federal personnel, the interagency agreement (IAA) established the scope of work and responsibilities for 
the participating federal agencies. Among other provisions, it outlines procedures for public outreach, pilot selection, the process for 
approving requests for waivers and blending of federal funds, pilot monitoring, and evaluations. It covers both general provisions and 
individual responsibilities for each agency. 


Table 1: Overview of the First Round Pilots 


Enrolled 
Project Site Youth Evaluation Description / Project Focus 


Baton Rouge Coordinates education, WIOA Title | Youth funds 
Broward County Combines after school, WIOA, library funds 
Chicago Coordinates Head Start, WIOA for teen mothers 
Indianapolis Public housing youth 

Los Angeles Promise Zone, system-wide effort 

Oklahoma City Foster youth aged 14-18 

Seattle-King County Coordinates WIOA Title I, reintegration, AmeriCorps 
Southeastern Kentucky Promise Zone, rural youth 

Ysleta del Sur Pueblo Native American youth aged 14-24 


Abbreviations 


RCT: randomized controlled trial 
QED: quasi-experimental design 


Source: Public record requests and interviews. For more detailed information, see Social Innovation 
Research Center, “P3 Project Summaries: 2015 Awards,” May 2016. Available at: 


http://socialinnovationcenter.org/wp-content/uploads/2016/05/P3 Project Summaries.pdf 


The Department of Education was assigned the role of lead federal administering agency. It published a formal 
notice inviting applications on November 24, 2014, offering a total of $7.1 million in start-up grants for up to 10 
pilot sites.2? Twenty-seven eligible entities applied in the following spring and nine were awarded grants in late 
September 2015, with a public announcement made in October.4 


Over the following months, the awardees and the relevant federal agencies negotiated the details of legally 
binding performance agreements. These agreements described the approved waivers of federal regulatory 
requirements, blending or braiding of federal funds, performance metrics, and enforcement mechanisms. Most 
of the performance agreements were completed and signed by late March 2016. 


The nine first round P3 pilots are summarized in Table 1. The projects vary substantially in size, with four 
serving fewer than 100 youth, but two others (Los Angeles and Southeastern Kentucky) focusing system-wide 
and serving a thousand youth or more. The two largest projects in Los Angeles and Kentucky are designated 
federal Promise Zones and thus eligible for competitive preferences under P3 and a number of other federal 
grant programs.” 


The projects also vary in the types of youth served and the services provided. While all nine serve youth who 
are disconnected or at risk of being disconnected, some are serving specific subpopulations, including rural 
youth (Southeastern Kentucky), tribal youth (Ysleta del Sur Pueblo), teen mothers (Chicago), foster youth 
(Oklahoma City), and youth located in public housing (Indianapolis). 


23 U.S. Department of Education, “Applications for New Awards; Performance Partnership Pilots,” Federal Register, November 24, 2014. 
Available at: https://www.federalregister.gov/articles/2014/11/24/2014-27775/applications-for-new-awards-performance-partnership-pilots 
U.S. Department of Education, "Obama Administration Names 9 Communities Chosen As Finalists for Pilot to Improve the Outcomes of 
Disconnected Youth,” October 29, 2016. See: http://www.ed.gov/news/press-releases/obama-administration-names-9-communities- 
chosen-finalists-pilot-improve-outcomes-disconnected-youth. 

U.S. Department of Housing and Urban Development, "Promise Zones." See 
http://portal.hud.gov/hudportal/HUD?src=/program_offices/comm_planning/economicdevelopment/programs/pz 
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Most of the approved flexibility requests, which may be amended and expanded, appear to be modest. Most of 
the nine pilots received no more than two waivers. Four were granted authority to blend federal funds. As 
required by law, all of these flexibilities granted under P3 authority are scheduled to expire on September 30, 
2018. 


The sites received one-time start-up grants, almost all of which were at or near the $700,000 maximum. The 
largest portion of these grants is being devoted to project management and evaluations. 


Most of the sites, at least to some extent, are relying on existing data systems to provide the outcomes data 
needed to track their performance. The sophistication and interoperability of these systems vary significantly. 


All nine have planned and budgeted for local evaluations, but only two involve randomized controlled trials 
(RCTs). The others are using quasi-experimental designs (QED), which will compare the outcomes for 
participating youth with other local youth with similar characteristics.2° 


Additional Competitions 


While this paper focuses on the experience of the first round pilots, Congress has passed legislation that 
authorizes additional pilots. This legislation also expanded the existing P3 authority to add certain discretionary 
programs at the Department of Justice 2” and HUD.8 


According to the administration, it expects to select up to 20 new pilots in two competitions in the coming year. 
On April 26, 2016, the administration announced the first of these two competitions with start-up grants of up to 
$350,000, or approximately half the size of the grants offered in the first round.29 9° 


The remainder of this paper reviews initial lessons from the first round projects across several common themes: 
(1) collaboration and coordinated services; (2) flexibility; (3) performance metrics; (4) data systems; (5) 
evaluations; and (6) technical assistance. 


Collaboration and Coordinated Services 


While flexibility is a major focus of the P3 program, so too is the promotion of collaborative state, local, and tribal 
partnerships. The importance of such collaboration is evident from the program’s name: Performance 
Partnership Pilots. 


The intersection between flexibility and collaboration is discernable at many levels. At the individual and family 

level, some of the focus is on wrap-around services, a holistic and comprehensive approach to serving children, 
youth, and families with complex needs.*! An alternative approach focuses not on comprehensive services, but 
appropriate services as determined through a needs assessments of youth and/or their families. This strategy is 
reflected in the phrase “bringing the right services to the right child at the right time.” ° 


26 Descriptions of the methodologies used, where available, can be found in the separate project summaries. See: 


http://socialinnovationcenter.org/wp-content/uploads/2016/05/P3 Project Summaries.pdf 

27 PL 113-235, which covered funding for federal fiscal year 2015, was signed into law on December 16, 2014. It continued earlier authority 
(Division G: Sec. 524.) and expanded it to cover funding for the Office of Justice Programs at DOJ (Division B: Sec. 219). This authority 
was extended in the following year as well (see next footnote). 

28 PL 114-113, which covered funding for fiscal year 2016, was signed into law on December 18, 2015. It continued earlier P3 authority 
(Division H: Sec. 525 and Division B: Sec. 219).and expanded it to include homeless assistance grants at HUD (Division L: Sec. 242). 

22 U.S. Department of Education, “Applications for New Awards; Performance Partnership Pilots,” Federal Register, April 26, 2016. 
Available at: httos://www.federalregister.gov/articles/2016/04/26/2016-09748/applications-for-new-awards-performance-partnership-pilots 

3° Congress did not authorize any additional funds for P3. Instead, it authorized the participating agencies to draw on other already existing 
appropriated funds. 

31 For an introduction, see National Wraparound Initiative, "Wraparound Basics." Available at: http://nwi.pdx.edu/wraparound-basics/. See 
also Interagency Forum on Disconnected Youth, “Summary of RFl Themes,” March 2013, p. 9. Available at: 
http://youth.gov/docs/Summary%200f%20RF|I%20themes.pdf 

82 Interviews with local P3 pilots. 


While comprehensive or tailored services can be provided by a single nonprofit or public agency, the needs of 
disconnected youth often require coordination among multiple public agencies and service providers. In recent 
years, this collaborative approach has taken hold in several fields, including education and social services, 
sometimes under the name “collective impact.” %° 


The focus on collective impact was present in much of the work and thinking that preceded the creation of P3, 
including the work of the White House Council for Community Solutions.*4 Collaboration was also emphasized 
in the P3 grant application materials, which noted that: 


Partnerships are critical to pilots’ ability to provide innovative and effective service-delivery and systems 
change strategies that meet the education, employment, and other needs of disconnected youth. We 
encourage applicants to build on strong, existing partnerships that have experience in working together 
to improve outcomes for disconnected youth. %5 


The administration does not characterize P3 as a collective impact-based program, but P3 shares much in 
common with (and can include) these efforts. Experts have cited several central components that are required 
for collective impact efforts to be successful. 9° Many of these are present in the local P3 projects, including: 


e Common Agenda: Disconnected youth often work with many different organizations, including those 
focusing on increasing high school graduation, workforce engagement, or related goals such as 
preventing teenage pregnancy. Most successful collective impact efforts unite such disparate efforts by 
creating partnerships with a common agenda and common set of goals. 


Such local partnerships are present to varying degrees in all of the P3 projects. The number of partners 
generally range from three to a dozen or more. Typical partners include schools, local workforce 
programs, other public agencies such as Head Start or housing agencies, community colleges, nonprofit 
social service providers, and outside researchers that are often based at local colleges or universities. 


In interviews, several of the P3 projects emphasized that it was important to start small and early, 
building on prior work and existing relationships, and then add partners over time. "We started small. 
That’s how we put our application together,” said one local project leader. "There was a long period of 
time between when the grant was announced and when we submitted. The additional time made our 
application better." 


e Shared Measurement: After agreement is reached on broad goals, the next step is to translate these 
goals into concrete, measureable indicators that will allow progress to be tracked and adjustments to be 
made over time. Such measures often draw on several existing data sources, each with its own privacy- 
related protections that must usually be addressed through consent forms and/or organizational data 
sharing agreements. 


While participating organizations may have their own individual measures, there is also a broad, single 
set of measures that is jointly shared by the project as a whole. For the P3 pilots, these measures are 
included in the performance agreements, which are discussed in greater detail later in this paper. 


e Mutually Reinforcing Activities: Making progress in collective impact-based efforts usually assumes 
coordinated, complementary efforts by the local partners. For P3, the interlocking nature of this work 


33 John Kania and Mark Kramer, "Collective Impact," Stanford Social Innovation Review, Winter 2011. See: 


http://ssir.org/articles/entry/collective impact . See also Jeffrey Henig et al., "Putting Collective Impact Into Context: A Review of the 
Literature on Local Cross-Sector Collaboration to Improve Education," October 2015. Available at: 
http://www.wallacefoundation.org/knowledge-center/summer-and-extended-learning-time/extended-learning-time/Pages/Putting- 
Collective-Impact-Into-Context.aspx 

White House Council for Community Solutions, "Community Solutions for Opportunity Youth: Final Report," June 2012. Available at: 
http://www.serve.gov/sites/default/files/ctools/12 0604whccs_finalreport.pdf 

U.S Department of Education, “Applications for New Awards; Performance Partnership Pilots,” Federal Register, November 24, 2014. 
Available at: https://www.federalregister.gov/articles/2014/11/24/2014-27775/applications-for-new-awards-performance-partnership-pilots 
This summary draws heavily on the framework outlined in FSG, "Collective Impact for Opportunity Youth," 2012. Available at: 
http://www.aspeninstitute.org/sites/default/files/content/docs/resources/FSG Collective Impact_for Opportunity Youth Report.pdf 
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was often expressed in logic models and theories of change that were included in the grant applications. 
They were also expressed more formally in local memoranda of understanding (MOUs) and in the 
federal performance agreements. 


"We have had great cooperation across a variety of partners," said Michael Twyman, Executive Director 
of OpportunIndy. "You don’t see that too often and it helped that the feds were working together, too. 
That is a smarter way for us to work. It allows us to bring more holistic services to individuals." 


¢ Continuous Communication: While responsibilities are often expressed in formal legal arrangements, 
such as local MOUs, the actual interactions among local partners are often more dynamic, requiring 
ongoing communication and adjustment that relied heavily on relationships and trust. 


“The trust level exhibited in our application is pretty far-reaching,” said Cindy Arenberg-Seltzer, President 
and CEO of the Children's Services Council of Broward County. 


"Government has a reputation for being rigid," she continued. "We pride ourselves for being a different 
kind of government that is responsive and best serves children and families. We have turned ourselves 
into pretzels to accommodate others and hopefully we will be really proud of the results. It would have 
been easy for us to walk away if we weren’t so committed.” 


e Backbone Support: Complex, collaborative efforts do not typically hold together without dedicated staff 
support. The lack of such support is one of the principal reasons why such efforts fail.2” Managerial or 
“backbone” staff typically serve a number of specific functions in collective impact efforts, including: (1) 
guiding vision and strategy, (2) brokering relationships to align activities, (3) establishing shared 
measurement practices, (4) building public support, (5) advancing policy, and (6) mobilizing funding.*® 


All of the P3 pilots have designated lead project managers to oversee these functions. Most drew 
heavily upon the P3 start-up grants to fund these positions.*9 


While these five components are important, at least two additional factors are also essential. One is resources. 
While collective impact strategies focus on aligning existing resources, there must be resources to align. In P3 
such resources are assumed, with pilots asked to coordinate services that have already been funded by other 
federal and non-federal programs. 


The P3 start-up grants have helped by gap-filling on important functions such as evaluations and backbone staff 
support.4° However, these one-time start-up funds — typically at or near $700,000 — were comparably small and 
often dwarfed by the larger funding streams that were being coordinated. When asked, all of the interviewed 
pilots said that if they were successful they would be able to find replacement resources to sustain their projects 
after the P3-funded phase ends. 


Another factor may be more problematic. In its review of successful community-based projects, the White House 
Council for Community Solutions identified longer-term investment as a critical factor for success. According to 
its final report, “Successful collaboratives make multi-year commitments because long-term change takes time. 
Even after meeting goals, a collaborative must work to sustain them.” * 


87 John Kania and Mark Kramer, "Collective Impact," Stanford Social Innovation Review, Winter 2011. See: 


http://ssir.org/articles/entry/collective impact 
38 FSG, "Collective Impact for Opportunity Youth," 2012, p. 34. Available at: 
http://www.aspeninstitute.org/sites/default/files/content/docs/resources/FSG Collective _Impact_for Opportunity Youth Report.pdf 
8° More information about successful backbone organizations and staff can be found in Shiloh Turner, Kathy Merchant, John Kania & Ellen 
Martin, "Understanding the Value of Backbone Organizations in Collective Impact," Stanford Social Innovation Review, July 17, 2012. 
Available at: http://ssir.org/articles/entry/understanding the value of backbone organizations in collective impact_1 
As noted earlier, Congress did not authorize any additional funds for P3. Instead, it authorized the participating agencies to draw on other 
existing appropriated funds. 
White House Council for Community Solutions, "Community Solutions for Opportunity Youth: Final Report," June 2012, p. 13. Available 
at: http://www.serve.gov/sites/default/files/ctools/12 O604whccs_finalreport.pdf 
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While P3 exhibits most of the qualities deemed necessary for success for collaborative or collective impact- 
based strategies, it may fall short on this measure. Local pilots may be able to continue on their own with other 
federal funding, but federal support specific to P3 (including waiver and blending authority and start-up grants) 
will end in 2018. 


Flexibility 


While local collaboration and partnerships are central to P3, they are not what make it unique. There are several 
federal initiatives that emphasize such partnerships, including Promise Zones, Full Service Community Schools, 
Promise Neighborhoods, and many others. What sets P3 apart from these other initiatives is its legal authority to 
align and/or waive certain legislative, regulatory, or administrative requirements that may prevent these 
partnerships from improving outcomes for disconnected youth. 


So far, the use of this authority appears to be modest. This section reviews this authority, its use thus far, and 
challenges that may be hindering greater utilization. 


Existing Waiver Authority Under P3 


P3 provides federal agencies the authority to waive any statutory, regulatory, or other administrative requirement 
under covered federal programs, subject to certain restrictions. For the first round, the law applies to annually 
appropriated programs operated by the five original participating agencies — the Departments of Education, 
Labor, and Health and Human Services, the Corporation for National and Community Service, and the Institute 
of Museum and Library Services. The law permits participating agencies to: (1) waive anything they are already 
permitted to waive under current law, and (2) approve other waivers that would not otherwise be allowed, subject 
to certain safeguards. 


The law contains a number of such safeguards. It excludes mandatory or entitlement spending (e.g., Medicaid, 
Social Security, Supplemental Nutrition Assistance Program (SNAP), most foster care IV—E programs, etc.). The 
waivers must be consistent with the overall statutory purpose of the target programs and also be necessary to 
achieve the purposes of the proposed pilot.42 They may not be used to waive nondiscrimination or wage and 
labor standards or to waive restrictions on the allocation of funds to states or other sub-state actors. 
Participating federal agencies must also determine that the proposed use of program funds will: (1) not result in 
denying or restricting individual eligibility for services funded by those programs; and (2) not adversely affect 
vulnerable populations that are the recipients of those services. 


The waivers are also temporary, with first-round flexibility measures expiring in 2018. They may be amended or 
revoked if the pilot is not achieving agreed upon outcomes or complying with applicable federal requirements. 


Subject to these restrictions, the local pilots may propose to waive any federal barriers that they think are 
necessary to achieve their program goals. According to the participating federal agencies, examples of potential 
requests for flexibility include, but are not limited to, changes to eligibility requirements, allowable uses of funds, 
and performance reporting.*2 Local projects may also propose to blend federal funds, a process of combining 
multiple funding streams, each with its own separate administrative requirements, into a single stream with a 
single set of administrative requirements.*4 


They must also result in efficiencies by simplifying reporting burdens or reducing administrative barriers with respect to such discretionary 
funds or increase the ability of individuals to obtain access to services that are provided by such discretionary funds. 

43 Webinar transcript, “Performance Partnership Pilots: FY 2014 Notice Inviting Applications,” December 1, 2014. 

For a good overview of blending and braiding strategies, see Ounce of Prevention, "Blending and Braiding Early Childhood Program 
Funding Streams Toolkit," November 2013. Available at: http://grisnetwork.org/resource/2013/blending-and-braiding-early-childhood- 


program-funding-streams-toolkit 
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According to administration officials, these flexibility provisions are intended to be bottom-up, driven from the 
local level. “Our intention is for this program to be truly driven from the community. We don't want to be too 
directive from the federal government, telling communities what envelopes they should be pushing,” said one 
federal official in an interview. 


Approved Waivers 


What waivers have been granted? As can be seen in Table 2, waivers have been approved for a variety of 
programs spread across the five participating federal agencies. The most widely affected program is the 
Workforce Innovation and Opportunity Act (WIOA) Title | Youth Program.*® 


Most of these waivers fall into one of the following categories: 


Blending and Braiding Federal Funds: Blending is a strategy that combines two or more funding 
streams, or portions of funding streams, into a single stream to produce greater efficiency. According to 
a 2015 description, "Funds from each individual stream lose their award-specific identity, and the 
blended funds together become subject to a single set of reporting and other requirements, consistent 
with the underlying purposes of the programs for which the funds were appropriated." 46 


Four of the nine pilots have received permission to blend federal funds.*” 


Braiding funds, by contrast, allows funding streams to be used for complementary purposes, but they 
must still be tracked and accounted for separately. Whereas blending requires one or more waivers 
from the appropriate federal agencies, braiding does not. However, waivers can facilitate more effective 
use of braided funds. 


Aligning Eligibility Rules: Many federal programs have different eligibility requirements (often tied to 
age or income eligibility), which hinder coordination. Several of the P3 pilots were granted waiver 
requests to align federal eligibility requirements. For example, the Baton Rouge Performance 
Partnership Pilot successfully requested a waiver of eligibility requirements under the Carl D. Perkins 
Career and Technical Education Act to allow services to youth starting in the sixth grade (the law 
otherwise prohibits the use of funds for students prior to the seventh grade). 


Aligning Performance Goals and Streamlining Reporting Requirements: Most federal programs 
have their own reporting requirements. Streamlining these reporting requirements can reduce paperwork 
and make it easier to provide a consistent set of services to targeted youth. 


Many of the first round sites received waivers that aligned the performance goals or reporting 
requirements for their component programs. Several replaced existing performance metrics with the 
single, common set of metrics established in their P3 performance agreements (described in the next 
section). 


Adjusting Federal Match Requirements: Projects located in low-income areas, including rural regions 
and tribes, can sometimes struggle with federal financial matching requirements. The Ysleta del Sur 
Pueblo, a federally recognized tribe, successfully requested a waiver of such matching requirements 
from the Corporation for National and Community Service. 


45 U.S. Department of Labor, "The Workforce Innovation and Opportunity Act: Youth Program Fact Sheet," March 2015. See: 
https://www.doleta.gov/wioa/Docs/WIOA _YouthProgram_FactSheet.pdf 


46 U.S Department of Education, "Proposed Priorities, Requirements, Definitions, and Selection Criteria—Performance Partnership Pilots 
for Disconnected Youth," Federal Register, October 22, 2015. Available at: https://www.federalregister.gov/articles/2015/10/22/2015- 
26965/proposed-priorities-requirements-definitions-and-selection-criteria-performance-partnership-pilots 


4” This figure is separate from the P3 start-up funding, which was drawn from the participating federal agency budgets and is a form of 
blended funding. 
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Table 2: Federal Programs with Approved Waivers 


The following table summarizes federal programs where the first round pilots received waivers or were 


authorized to blend federal project funds. 


The table also includes programs identified in the FY 2014 Notice Inviting Applications as potentially eligible 
for inclusion in the pilots, including those where waivers have not yet been approved. This list is not 
comprehensive. Other programs not listed here may also be eligible for waivers under P3. 


Federal Agency / Program Pilot Project 


Corporation for National and Community Service 
e AmeriCorps 
e Social Innovation Fund 


Department of Education 
21st Century Community Learning Centers 
Career, Technical and Adult Education 
ESEA, Title I, Part D 
Full Service Community Schools 
GEAR UP 
General Admin. Regulations (EDGAR) 
Promise Neighborhoods 


Department of Health and Human Services 
Adolescent Pregnancy Prevention * 
Chafee Adult Education and Training 
Head Start 
Now Is the Time — Healthy Transitions 
Runaway and Homeless Youth * 
SAMHSA - various programs * 


* 


Department of Labor 
e Reintegration of Ex-offenders * 
e WIOA- Title | Adult 
e WIOA- Title | Youth 
e §=YouthBuild 


Institute of Museum and Library Services 
e Library Services and Technology Act 
e Native American and Library Services 
Enhancement Grants 


* 


Ysleta del Sur Pueblo; Ok City, Seattle (braiding) 
None 


Broward County 

Baton Rouge 

Baton Rouge (braiding) 
Southeastern Kentucky (braiding) 
Southeastern Kentucky 

Broward County, Chicago 
Southeastern Kentucky 


None 

None 

Chicago 

Oklahoma City (braiding) 
Los Angeles 

None 


Seattle 

None 

All except Ysleta del Sur Pueblo 
Indianapolis 


Broward County (braiding) 
Ysleta del Sur Pueblo 


Rated by the administration in the FY 2014 Notice Inviting Applications as requiring significant review to 


ensure that vulnerable populations are not adversely affected. 


Sources: Public record requests, interviews, and FY 2014 Notice Inviting Applications. For more detailed 
information, see Social Innovation Research Center, “P3 Project Summaries: 2015 Awards,” May 2016. 
Available at: http://socialinnovationcenter.org/wp-content/uploads/2016/05/P3 Project_Summaries.pdf 


e Funding New Services: While many of the ideas listed above make it easier to align existing services, 
another possible use of P3 flexibility is to repurpose existing funds for a new and innovative use that 
might not otherwise be allowed under existing rules. 


An example can be found in the Chicago Young Parents Program (CYPP), which designed a program for 
teen mothers using a combination of waivers from three federal departments. Similarly, the Broward 
County site obtained a waiver allowing them to use 21st Century Community Learning Center (after 
school) funds for activities during the school day. 


Flexibility Challenges 


So far, most of the approved waivers appear to be modest, both in numbers and in scope. As noted earlier, only 
four of the nine pilots have been granted authority to blend federal funds. Most of the pilots have no more than 
two approved waivers and most of the approved waivers have been modest, involving what appear to be minor 
changes in program eligibility and reporting requirements. 


Administration officials disagree with this assessment. “A pilot may have only requested one or two waivers, but 
the waivers could be significant for that pilot,” said one federal official. “For example, a waiver of statutory 
performance measures offers the pilot an opportunity to tailor workforce and education measures to 
appropriately fit the goals of the program.” She indicated that early conclusions that characterize waiver impact 
are premature and require a more detailed evaluation.*® 


However, while the approved flexibility changes may be significant in those cases where new services are being 
piloted (described above) and/or where substantial blending has been authorized, this level of flexibility has not 
yet been approved for most of the pilots.*9 


There are a number of possible reasons for this: 


e Limited Local Knowledge of Federal Barriers: One operating assumption of the P3 program appears 
to be that it should be bottom-up, with local communities driving the flexibility requests.5° Local program 
coordinators are assumed to be best positioned to identify barriers that are hindering their success. 


In fact, where they existed, there was considerable confusion at the local level about the origin of certain 
barriers. In some cases, states and other authorities had been blocking local requests for flexibility by 
arguing that they were prohibited by federal law. P3 allowed these local organizations to press through 
such objections. Often when they did, they found that the objections were unfounded. 


These efforts revealed what was perhaps an unintended, but positive, consequence of P3. Even when 
waivers were not granted (because they were unnecessary), the process still increased local flexibility. 


Nevertheless, limited local knowledge of federal restrictions remains a significant barrier. In interviews, 
several of the pilots suggested that they were eager to learn from the other projects and that they 
welcomed federal technical assistance that would help them identify additional barriers. Several 
suggested that it would likely be an iterative / learning process, with considerable improvement being 
made over time. 


e Recent Changes in Other Federal Laws: One potentially complicating factor for P3 was that at the 
time that it was being designed and implemented, two other major federal laws with significant 
implications for P3 were being amended by Congress. The Workforce Innovation and Opportunity Act, a 


Email communications, May 13 and May 18, 2016. 

48 Complete descriptions of all of the waivers can be found in Social Innovation Research Center, “P3 Project Summaries: 2015 Awards,” 
May 2016. Available at: http://socialinnovationcenter.org/wp-content/uploads/2016/05/P3_Project_ Summaries.pdf 

This may be partly to prevent a recurrence of the negative reactions that were generated when the Obama administration used state- 
level waivers under No Child Left Behind to promote administration-approved education reforms. 
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significant update of federal workforce and training programs, was enacted in 2014.5' The Every 
Student Succeeds Act, a major rewrite of the nation’s primary K-12 education law, was enacted in late 
2015.52 These changes have left the nation’s workforce and education laws in substantial flux. With the 
administration currently devising regulations for both laws, they represent a moving target for any 
flexibility requests under P3. In interviews, federal officials indicated that the pilots would receive 
technical assistance on changes to these laws and regulations that affect their projects. 


e State and Local Barriers: Federal restrictions are not the only barriers to local flexibility and improved 
performance. In many cases, such restrictions may originate at the state, local, or tribal level. P3 does 
not directly address these non-federal barriers. Instead, it relies on the pilots to work with state and local 
authorities to address them when needed. 


e Limited Time: By law, authority for the first round projects expires on September 30, 2018.5° While the 
administration moved comparably quickly, it still took time to launch P3 and this placed added pressure 
on the local pilots and federal officials to finalize waiver agreements as quickly as possible. 


Such time pressures may have limited the opportunity to secure more than the easiest and most 
essential waivers. There appears to be some presumption among at least some of the pilots that their 
initial list of approved waivers will be expanded. 


e Restrictive Federal Safeguards: Some of the safeguards that were put in place by Congress to 
prevent abuse may have prevented the approval of certain flexibility and blending requests. The 
administration noted in the FY 2014 Notice Inviting Applications that some federal programs were more 
likely to pose challenges for these reasons.** The administration has provided further details about the 
legal limits placed on local flexibility requests in associated Frequently Asked Questions documents.°> 


Many of these barriers seem likely to be overcome over time, particularly as local and federal authorities learn 
more and test the limits of their new authority. However, there is at least one additional barrier that may require 
congressional action: 


e Waiver Expiration: As noted above, by law the waiver authority for all of the first round pilots expires in 
2018. (Waivers granted for second and third round projects will expire in 2019 and 2020, respectively). 


Given the time required to launch the projects, along with a presumption in the literature that successful 
collaborative partnership strategies require a long-term, multi-year commitment,°* these expiration dates 
may be problematic, particularly for pilots with promising early results. Congress may wish to revisit 
these deadlines in future legislation. 


Performance Metrics 

At least in part, the P3 program is a test of the idea that reduced regulatory accountability can be exchanged for 
an increase in performance-based accountability. While increased flexibility and coordinated services may be 
the means, improved outcomes for youth are the intended end. 


To help track their progress toward this goal, each project's performance agreement includes project-wide 


See https://www.doleta.gov/wioa/ 

See http://www.ed.gov/essa 

53 PL 113-76, Division H: Sec. 526(c)(2)(A) 

U.S Department of Education, “Applications for New Awards; Performance Partnership Pilots,” Federal Register, November 24, 2014. 
See Appendix B: Examples of Programs Potentially Eligible for Inclusion in Pilots. Available at: 
https://www.federalregister.gov/articles/2014/11/24/2014-27775/applications-for-new-awards-performance-partnership-pilots 

55 Youth.gov,"Performance Partnership Pilots (P3) Round 2 (FY 2015) Frequently Asked Questions," May 4, 2016. See Section C 
(Waivers). Available at: http://youth.gov/youth-topics/reconnecting-youth/performance-partnership-pilots/faq 

White House Council for Community Solutions, "Community Solutions for Opportunity Youth: Final Report," June 2012, pp. 13, 25, 30, 
39. Available at: http://www.serve.gov/sites/default/files/ctools/12 O604whccs_finalreport.pdf 
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metrics that track educational, employment and other key outcomes. Examples from the performance 
agreements include the following: 5” 


e Education Metrics: Education-related metrics commonly include school attendance, educational 
achievement, high school graduation rates, GED high school equivalency, and participation in post- 
secondary education. 


e Employment Metrics: Employment metrics commonly cover skill development, program completion, 
attainment of career-related certifications, and employment. 


e Other Metrics: Other metrics include rates of stabilized housing, reduced involvement in the criminal 
justice system, and improved physical and mental health. Some P3 projects are also tracking system- 
level changes, such as meeting specified milestones for collaboration and data sharing. 


According to administration officials, the metrics were initially proposed by the local projects and were often 
drawn from their grant applications. The final metrics were negotiated between the projects and the relevant 
federal agencies. Administration officials said they sought to respect local prerogatives consistent with the 
requirements of the law. 


The final negotiated metrics typically include a range of interim, process, and outcome measures. In some 
cases, the metrics appear to be interrelated and rooted in varying points of the project's logic model.5® Such 
diverse measures facilitate not just the tracking of progress along milestones, but also diagnosing and 
addressing potential problems when they arise. 


The performance agreements also specify data sources and methodology. In most cases, they include numerical 
performance targets (for example, “80% of participants will pass state mandated end of course exams”). Where 
possible, these targets have been tied to existing baseline information for the projects. In other cases where 
such information is unavailable, best professional judgment was used, but local pilots agreed to collect baseline 
information. Federal officials sought to ensure that the agreements captured improvements in performance, as 
required by the authorizing legislation.°9 They also suggested that they were sensitive to the need to choose 
measures and targets that did not discourage the pilots from serving the hardest-to-serve youth.®° 


Finally, each performance agreement includes provisions for addressing performance shortfalls. These include a 
range of possible options that can be invoked by the participating federal agencies as needed, including: 
providing technical assistance to the pilots; amending, revoking, or granting additional waivers; requiring 
corrective action plans; requiring the repayment of misspent funds; and amending or terminating the 
performance agreement. 


Data Systems 


While the P3 program relies heavily on performance data, it provides no dedicated funding to cover these costs. 
Most of the first round pilots are using comparably little of their start-up funding for these purposes. Instead, local 
projects are relying heavily on direct or indirect access to data infrastructures that already exist. 


The expectation that the pilots would bring such pre-existing capacity was built into the 2014 P3 grant 
application. It reserved the largest share of points (30 out of up to 112) for applicants that could demonstrate the 
ability to collect and use such data, including executing data-sharing agreements, managing and linking data, 
maintaining data quality, and protecting privacy. 


Details on performance metrics for each P3 project can be found in Social Innovation Research Center, “P3 Project Summaries: 2015 
Awards,” April 2016. Available at: http://socialinnovationcenter.org/wp-content/uploads/2016/05/P3_Project_ Summaries.pdf 

For more information on logic models, see: Whitebarn Consulting, "Resources: Logic Models and Theories of Change (updated)," 
December 1, 2015. Available at: http://www.whitebarn.info/s/resources-logic-models-and-theories-of-change 

59 PL 113-76, Division H: Sec. 526 (a)(2) and Sec. 526 (b)(1). 

60 Interview, March 1, 2016. 
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Despite this expectation, there remains significant variation among the grantees. In some cases, the local project 
leads are relying on pre-existing data systems that they established and operate themselves. In other cases, 
they have memoranda of understanding in place with project partners and are indirectly accessing the needed 
data through periodic downloads. 


For example, the Los Angeles Performance Partnership Pilot is primarily relying on the city’s workforce data 
management system, with additional data being accessed from the Los Angeles Unified School District through 
a pre-existing agreement. The Chicago Young Parents Program is relying primarily on its pre-existing Head 
Start data system, with additional data accessed from its principal nonprofit partner, which is a Head Start 
provider. 


In other cases, even where data sharing agreements are already in place, the level of direct interoperability 
between the systems is often mixed. In many cases, particularly the smaller projects or those working with new 
partner organizations, data is being downloaded, matched, and combined manually, sometimes in new 
databases that have been created specifically for the P3 project. 


When limited P3 funding is being spent on data, it usually only covers marginal costs, such as additional 
software licenses, training, or part-time support from a data administrator. If these projects are sustained or 
scaled up after federal P3 funding ends, they will need to find alternative funding for more fully interoperable 
systems. 


Evaluations 


Evaluations, both local and national, are a major part of P3. However, while the program is expected to make 
significant contributions, the evidence bases for programs addressing disconnected youth in general and 
collaborative partnership-based strategies in particular are underdeveloped.®' Given the nascent state of the 
field, building this evidence base will take time. 


As currently designed, the individual evaluations for the local pilots vary in their design and rigor. Some may 
produce significant results that will add to the evidence base for disconnected youth, while others may be 
preliminary. The national evaluation of the first round of pilots also is not designed to draw definitive conclusions 
about the program’s overall impact on disconnected youth. Instead it will focus on documenting and assessing 
the extent of system change, capacity building, and partnership development. 


There are several reasons for this. 


e Local Evaluations Currently Vary in their Design and Rigor, although the Evidence Bar May Rise 
over Time: Each of the pilots is conducting its own evaluation. In most cases, these evaluations are 
being conducted by local academics or evaluation firms with knowledge of the region or project. In a few 
cases the projects are using non-local evaluators with specific experience in certain topics, such as 
collaborative partnership-based strategies. 


Local evaluation budgets commonly range from $100,000 to $200,000 and draw heavily upon the P3 
start-up grants.62 The pilots are also receiving technical assistance on their evaluations from the 
program’s national evaluator, Mathematica Policy Research. 


Although the federal grant application for the first year’s cohort provided an incentive to conduct a 


61 MDRC, "What Works for Disconnected Young People: A Scan of the Evidence," February 2016, pp. 26-29. Available at: 
http://www.mdrc.org/publication/what-works-disconnected-young-people See also MDRC, "Serving Out-of-School Youth Under the 
Workforce Innovation and Opportunity Act," June 2015. Available at: http://www.mdrc.org/sites/default/files/Serving Out-of- 

School Youth 2015%20NEW.pdf 

62 These figures may be in line with similar evaluation costs under the Social Innovation Fund. See: Corporation for National and 

Community Service, Office of Research and Evaluation, "Budgeting for Rigorous Evaluation: Insights from the Social Innovation Fund,” 


2013, p. 23. Available at: http://www.nationalservice.gov/sites/default/files/documents/Budgeting for Evaluation.pdf 
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randomized controlled trial (RCT) based study,® often referred to as the “gold standard” of evaluation, 
only two of the first nine projects are planning to conduct such studies (Baton Rouge and Broward 
County). The rest will be using quasi-experimental designs that will compare the results for participating 
youth to other youth with similar characteristics. 


Federal officials say that RCT-based studies should only be used when they are well-designed and 
appropriate for the project, including a sufficient number of program participants for statistical precision 
and stable program implementation. According to the Department of Labor, the second round 
competition continues to provide competitive points to applicants that propose impact or outcome 
evaluations, but will not limit the extra points specifically to RCT-based experimental evaluations.® 
Nevertheless, administration officials believe that the new criteria will still incentivize RCTs because of 
their associated rigor, which remains a factor in the scoring. 


"We believe that this approach balances that incentive with the need to address high-value research 
questions using the most appropriate design, which may or may not involve random assignment for a 
given pilot," said one administration official. 6 


"One of the provisions of the national evaluation is that if any of the grantees are doing something really 
innovative that is not being rigorously evaluated by the local evaluator, we have options to do more 
formal RCT studies,” said another federal official.6” 


The level of rigor may increase further in the coming years. While the evidence base is still young and 
unevenly distributed, advances are occurring across the field. The What Works Clearinghouse at the 
U.S. Department of Education and the Clearinghouse for Labor Evaluation and Research (CLEAR) at 
the U.S. Department of Labor are both cataloguing the growing body of evidence for education and 
workforce programs, two critical components of most P3 initiatives. Collective impact-based approaches 
to disconnected youth are also receiving increased evaluation attention.® 


As the field advances, the evidence bar for P3 may rise, especially if more organizations with experience 
in evidence-based programs apply and are selected. If the number of pilots and funding are expanded, 
federal policymakers may wish to institute a tiered evidence approach similar to that used in the i3 
program at the Department of Education, which would allow the program to fund different initiatives at 
different stages along the evidence spectrum. 


e In Some Cases, Final Evaluations of Local Pilots May Come Too Early to Provide Conclusive 
Results: At the local level, each of the projects has many moving parts, with multiple partners providing 
different services to local youth. Most of the projects are relatively new and will take time to become 
fully operational. Federal grants of flexibility may also change during the 2- to 3-year project windows, 
which may present a moving target for evaluators (although federal officials say they will work with 
evaluators to ensure that mid-course corrections do not interfere with evaluation efforts).® 


Most of the local evaluations will include interim studies to help guide implementation. However, in some 
cases even the final evaluations may come too soon to provide more than an early snapshot of program 
effectiveness for projects that may take several years to reach their potential. 


U.S. Department of Education, “Applications for New Awards; Performance Partnership Pilots,” Federal Register, November 24, 2014. 
Available at: https://www.federalregister.gov/articles/2014/11/24/2014-27775/applications-for-new-awards-performance-partnership-pilots 


Interview, April 5, 2016. 

See also U.S. Department of Education, "Proposed Priorities, Requirements, Definitions, and Selection Criteria—Performance 
Partnership Pilots for Disconnected Youth," Federal Register, October 22, 2015. Available at: 
https://www.federalregister.gov/articles/2015/10/22/2015-26965/proposed-priorities-requirements-definitions-and-selection-criteria- 


performance-partnership-pilots 

Email communication, May 13, 2016. 

Interview, April 5, 2016. 

MDRC, "What Works for Disconnected Young People: A Scan of the Evidence," February 2016, pp. 26-29. Available at: 
http://www.mdrc.org/publication/what-works-disconnected-young-people 

Email communication, May 13, 2016. 
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Some of the projects are substantially rooted in previous work, however, and they may be better 
positioned to produce conclusive results. The Chicago P3 project, for example, is based on an earlier 
program that was piloted in 2014 and evaluated by Chapin Hall at the University of Chicago. 


When interviewed, several local project leaders said that near-term trend data and qualitative 
evaluations would provide important insights on aspects of their initiatives that are working or that need 
mid-course corrections. Some suggested that they wanted to investigate longer-term effects, but would 
need to find outside funding for such studies because they would occur after P3 funding for their projects 
ends. 


"We will see some things in a few months. If you are really understanding what the young person needs, 
you will see attendance improve," said Cindy Arenberg-Seltzer of the Children's Services Council of 
Broward County. "For other things, it will take the full two years if not longer to see the changes." 


The National Evaluation Will Not Draw Definitive Conclusions about the Program’s Direct Impact 
on Youth in the Near Term: The Department of Labor is overseeing the national evaluation of P3, 
which is being conducted by Mathematica Policy Research. This early evaluation, which has already 
begun, is not designed to draw definitive conclusions about the program’s effectiveness. 


According to federal officials, the national evaluation will focus not on the program’s overall impact, but 
on documenting and analyzing systems-level changes such as improved local coordination, blending 
and braiding of funding, removal of organizational and administrative barriers to supporting disconnected 
youth, and building data capacity.” 


Technical Assistance 


While technical assistance is important for many programs, for a program as complicated and demanding as P3 
it may be especially so. According to the Department of Education, based on an assessment of the technical 
assistance needs of the nine pilots, it appears that the pilots would benefit from technical assistance on the 
following topics: 


Asset Mapping: Mapping related cross-sector or opportunity youth initiatives in their communities to 
determine areas for strategic alignment based on shared partners, objectives, target population, or 
outcomes; 


Flexibility: Maximizing the flexibility of waivers and the strategic use of braided funding from other 
federal and non-federal programs for opportunity youth; 


Accountability: Better defining roles, expectations, and deliverables for partners and holding partners 
accountable for outcomes; 


Youth Engagement: Developing strategies for youth leadership and engagement in program design 
and in addressing concerns impacting opportunity youth in the community; and 


Data and Evaluation: Clarifying data collection and evaluation plans as they relate to participant 
recruitment and enrollment, as well as the length and intensity of participants’ exposure to the 
intervention. 


The program has three technical assistance providers under contract, including Jobs for the Future and the 
Forum for Youth Investment (under subcontract). Mathematica Policy Research is providing assistance for 
evaluations. 


70 


Interview, April 5, 2016. Follow up email communication, May 13 2016. 
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All of the P3 pilots are also required to participate in a joint Community of Practice. The first Community of 
Practice meeting will be held June 20-21 in Washington D.C. 


Recommendations 


The P3 program holds significant promise for better aligning existing federal programs for disconnected youth 
and, under ideal circumstances, supporting innovative new solutions that may achieve better outcomes. More 
broadly, it may serve as a model for other populations beyond disconnected youth and other program areas 
where greater flexibility and cross-sector collaboration could produce better results. 


Fulfilling this potential could be made easier, however, if policymakers considered a number of supportive 
changes: 


e Federal Officials and Congress Should Support More Aggressive Use of P3’s Waiver Authority: 
Authority to waive federal restrictions and to blend federal funds is what sets P3 apart from other federal 
programs. So far, this authority does not appear to have been used as aggressively as it could be. 


While there appears to be an expectation that the pilots will take the lead on identifying barriers and 
requesting waivers, they need additional assistance. Federal officials should prioritize this in their 
technical assistance efforts. 


Legislatively-imposed safeguards, while well-intended, may also be overly broad and limiting. The 
safeguards are intended to protect vulnerable populations, but disconnected youth are themselves 
among the nation’s most disadvantaged and vulnerable groups. Congress and the administration 
should solicit input from stakeholders on these issues and consider modifying the existing safeguards to 
allow greater flexibility. 


e Congress Should Extend the Time Allowed for P3 Projects: Under current law, authority for all of 
the first round pilots expires on September 30, 2018. Authority for second and third round projects will 
expire in 2019 and 2020, respectively. While the projects may continue with local funding after those 
deadlines, the associated federal waivers will end. 


One consistent theme that has emerged in this review is the need for a longer-term commitment to local 
projects. The literature on collective impact, including the recommendations of the White House Council 
for Community Solutions, reinforces the need for a long-term perspective. By comparison, the amount 
of time being granted to local P3 projects is less than that being granted under other evidence-based 
initiatives like the Social Innovation Fund, where projects typically take five years or more.”! 


For P3 to provide meaningful insights on the underlying concepts of collaboration and increased 
flexibility, the P3 projects must be given enough time to reach their potential. While such authority need 
not be granted in perpetuity, waivers could be granted for five years with the option to apply for 
extensions. 


e The Evidence and Evaluation Requirements for P3 Should Be Strengthened: The evidence base 
for programs affecting disconnected youth is still in its infancy, but the evidence bar should rise over 
time. As the knowledge base grows, both the incoming evidence requirements and evaluation 
expectations should increase. 


™ Social Innovation Research Center, "Report: Social Innovation Fund’s Early Results Are Promising," June 30, 2015. Available at: 
http://www.socialinnovationcenter.org/?p=1413. See also Social Innovation Research Center, "Foster Care Innovation Initiative Charts a 
Different Path to Evidence," December 6, 2014. Available at: http://www.socialinnovationcenter.org/?p=769 
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Although the program’s national evaluation does not now include plans to examine the impact of the 
program as a whole on participating youth, this should change as the program becomes better 
established. 


e Federal Officials Should Promote Better Alignment between P3 and Other Federal Collaborative 
and Evidence-based Initiatives: |n its first round, P3 drew only 27 eligible applications. A larger 
applicant pool would allow it to choose stronger pilots in future competitions. To accomplish this, the 
program may wish to target potential applicants that are participating in other federal programs that are 
well-aligned with P3. The benefits of better alignment could also extend to joint technical assistance in 
some cases. 


With its focus on evidence-based programs and collaborative partnerships, the P3 program is well 
aligned with other federal programs such as Promise Zones, Promise Neighborhoods, Choice 
Neighborhoods, and Full Service Community Schools. It is also well-aligned with other evidence- 
focused initiatives, such as the Social Innovation Fund at CNCS and the Investing in Innovation (i3) 
program at the Department of Education.’ All of these programs fall within the jurisdictions of federal 
departments and agencies that are already participating in P3. 


Applicants that are part of federally-designated Promise Zones already receive a competitive preference. 
The administration may wish to provide similar preferences for other federal programs that are aligned 
with P3’s goals. 


—-o-— 


About the Social Innovation Research Center: The Social Innovation Research Center (SIRC) is a 
nonpartisan nonprofit research organization focused on social innovation and performance management for 
nonprofits and public agencies. More information about SIRC is available on the organization's web site at 


http ://www.socialinnovationcenter.org. 


This paper was developed with the generous financial support of the Laura and John Arnold Foundation. The 
opinions expressed in this paper are the author’s and do not necessarily represent the view of the foundation. 


72 More information about tiered evidence initiatives can be found on the Youth.gov web site at: http://youth.gov/evidence- 
innovation/investing-evidence 
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